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Introduction 

Within primary and secondary school technology education, engineering 
has been proposed as an avenue to bring about technological literacy (Lewis, 
2005; Dearing & Daugherty, 2004). Different initiatives such as curriculum 
development projects (i.e.. Project ProBase and Project Lead The Way) and 
National Science Foundation funded projects such as the National Center for 
Engineering and Technology Education (NCETE) have been developed to 
infuse engineering into primary and secondary education. For example, one key 
goal of the technology teacher education component of NCETE is to impact the 
focus and content of the technology education field at the secondary level 
(National Center for Engineering and Technology Education, 2005). More 
specifically, the goal is to facilitate students’ learning relative to core 
engineering principles, concepts, and ideas. A number of activities have been 
developed by the Center to facilitate these goals, including a series of teacher 
professional development experiences, research designed to identify core 
engineering concepts, development of engineering-related activities, 
engagement with faculty from the STEM disciplines, and interaction with 
technology education pre-service teachers. 

Through the efforts of NCETE, three core engineering concepts within the 
realm of engineering design have emerged as crucial areas of need within 
secondary level technology education. These concepts are constraints, 
optimization, and predictive analysis (COPA). COPA appears to be at the core 
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of the conceptual knowledge needed for students to understand and be able to 
do engineering design. 

One of NCETE’s five technology teacher professional development 
institutions (Illinois State University) has focused exclusively on the delivery of 
these COPA concepts. These concepts have emerged as distinctly important, 
based on analysis of multiple engineering design processes and the 
technological design and problem solving process. At Illinois State University, 
two cohorts of practicing and pre-service technology teachers have engaged in 
professional development workshops to become better prepared to deliver 
engineering concepts to their students. The three core engineering concepts, 
mentioned earlier, were identified: constraints, optimization, and predictive 
analysis. These were selected based on over three consecutive years of 
professional development experiences with teachers, partnerships with the 
engineering community, hands-on activities, and an analysis of related research. 
The review of prior research concentrated primarily on the nature of engineering 
and engineering design, how it differs from technology education processes, and 
the necessary conceptual and procedural knowledge. 

Empirical knowledge is needed to better understand how to increase student 
learning of COPA. This study sought to provide that knowledge using a NCETE 
cohort of practicing and pre-service technology teachers who designed and 
developed a unit of instruction to deliver these three core engineering concepts 
to secondary level technology education students. Using a mixed method quasi- 
experimental, pretest, post-test, no control group design, this study explored the 
extent to which students understood and were able to demonstrate an 
understanding of constraints, optimization, and predictive analysis. It is believed 
that through this strong conceptual base, a better understanding of engineering 
and engineering design can be achieved. 

Review of the Literature 


Predictive Analysis 

In a review of science, engineering and technology careers. Deal (1994) 
stated that engineers apply mathematical and scientific principles to solve 
problems. The introduction of these tools into the analytical stage of the design 
process represents an indispensable part of engineering design (Harris, & 
Jacobs, 1995). Eekels (1995) observed how the prediction component functions 
in the engineering design process, noting that “if the conditional prediction 
sounds unfavorable, then we generally simply abstain from that action and 
design another action,” (p. 176) which is to make the informed decision before 
constructing the prototype of a design. Hayes (1989) observed that predictive 
analysis is carried out in the planning environment, not the task environment, 
with several distinct advantages: (a) moves made in the planning environment 
can be easily undone while task environment actions cannot be reversed; (b) 
predictive analysis is relatively inexpensive; and (c) it permits design flexibility. 
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“The process of thinking before acting” is critical if designing is to be a 
predictive rather than a trial-and-error process (Hayes, 1989). Trial-and-error 
remains the prevailing approach to design in technology education classrooms, 
where analytical mathematical tools are frequently not used to design and 
prototype design ideas (Lewis, 1999; Merrill, 2001). Lewis (2005) argued that 
conceptual design is within the normal purview of technology education and 
that science and mathematics should be taught to help students make predictions 
about the design through the process of analytical design. The Standards for 
Technological Literacy: Content for the Study of Technology (International 
Technology Education Association, 2000) reinforce the systematic aspects of 
predictive analysis. “Because so many different designs and approaches exist to 
solving a problem, a designer is required to be systematic or else face the 
prospect of wandering endlessly in search of a solution” (International 
Technology Education Association, 2000, p. 91). 

Constraints 

The design processes utilized in engineering and technology education are 
very similar with some notable exceptions. Lewis (2005) has suggested that 
engineering design places more emphasis on assessing constraints, trade-offs, 
and utilizing predictive analysis compared to technology education. The 
importance of constraints is, however, included in the Standards for 
Technological Literacy: Content for the Study of Technology > (International 
Technology Education Association, 2000). In Standard Eight, constraints are 
viewed as an integral part of an iterative process that typically requires students 
to consider costs, economics, feasibility, time, material, and environmental 
implications. Students should be able to assess and incorporate constraints into 
design activities. 

Addressing constraints early in the problem identification stage may assist 
students in developing viable solutions, since this process helps reduce the size 
of the solution space (Jin & Chusilp, 2006). Expert designers typically move 
quickly from defining the problem (problem space) to the solution space by 
assessing the constraints to the problem and searching for contextually related 
problems that they have solved in the past (Cross, 2002; Cross, 2004; 

Middleton, 2005). This is similar to a model revised by Middleton, where the 
problem space is defined as the problem state, goal state, and search state. 
Middleton's “search state” can be viewed as identifying the constraints; while 
iteratively moving between the problem state and goal state, and concurrently 
decomposing an ill-defined problem into well-defined sub-problems (Cross, 
2002; Ho, 2001). 

Optimization 

Design optimization extends beyond simply producing a design that 
adheres to a defined set of constraints or criteria. The purpose of optimization is 
to achieve the “best” design relative to a set of prioritized criteria or constraints. 
These include maximizing factors such as productivity, strength, reliability. 
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longevity, efficiency, and utilization. Engineers must make many technological 
and managerial decisions during the design process in order to produce the best 
design. The ultimate goal of all such decisions is to minimize undesirable 
effects, while maximizing desirable effects, producing a “better, more efficient, 
less expensive solution that is in harmony with the laws of man and nature” 
(Ertas & Jones, 1993). 

Optimization typically occurs during the formulation of a design problem. 
According to Arora (1989), formulation of a problem requires approximately 
50% of the total effort needed to solve it. Optimization techniques provide well- 
defined procedures to aid the designer in correctly formulating the problem. For 
example, Statnikov (1999) outlined three questions that designers should be able 
to answer when formulating a design problem: 

1 . What to search for? (resulting in identifying the performance criteria.) 

2. Where to search? (resulting in defining all the constraints imposed on the 
design, which produces a set of feasible solutions.) 

3. How to search? (resulting in identifying the optimization technique that is 
most suited for the specific features of the problem being solved.) 

Formulating a design problem to achieve an optimal solution often involves 
transcribing a verbal description of the problem into a well-defined 
mathematical statement. This process enables the designer to search for the 
optimal design according to the identified performance criteria. Optimization 
methods frequently use mathematical concepts such as vector and matrix 
algebra, and calculus to analyze and optimize variables. As Arora (1989) 
pointed out, “the importance of proper formulation of a design optimization 
problem must be clearly understood because the optimum solution will only be 
as good as the formulation is” (p. 21). 

Purpose and Research Questions 

The purpose of this study was to assess the effectiveness of a unit of 
instruction in teaching core engineering concepts to secondary level technology 
education students. The following research questions guided the study. 

1 . What type of engineering design activities and lessons will effectively 
deliver selected core engineering concepts to technology education 
students at the 10-12 grade level? 

2. Is there a relationship between performance in mathematics courses taken 
prior to participating in the unit of instruction and post-test instructional 
gain? 

3. Is there a relationship between performance in physical science courses 
taken prior to participating in the unit of instruction and post-test 
instructional gain? 


Methodology 


Research Design 
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This study used a mixed methods quasi-experimental, pre-test post-test, no 
control group design, with the treatment as the independent variable and pretest- 
posttest as dependent variables. The participants in this study received a pre-test, 
treatment, and a post-test. The researchers chose not to use a control group in 
this study for several reasons. First, since the data were gathered at multiple 
high school locations, it would have been extremely difficult to apply a uniform 
control group experience across all schools (i.e., each school’s typical 
technology education curricula are different). Second, the logistics of 
identifying sufficient numbers of like courses taught by the same professionally 
developed teachers were problematic. While a control group design would have 
strengthened the study, the decision was made to proceed with a pre-test, post- 
test design augmented by a qualitative component, given the exploratory nature 
of the research. Additionally, after the posttest, focus groups with randomly 
selected participants were conducted to “stimulate embellished descriptions” 
(Fontana & Frey, 2005, p. 704) of the 20-class session unit of instruction. 

Treatment 

During the 2005-2006 school year, each technology education teacher who 
was involved in this study, as well as one mathematics and one science teacher, 
completed 120 hours of professional development related to infusing 
engineering concepts into high school technology education. During this 
professional development, the eight technology education, one mathematics, and 
one science teacher helped to develop the 20-class session unit of instruction 
and the activities that supported infusing engineering concepts into the 
curriculum. These teachers were chosen because they were already participating 
in the NCETE professional development from which the study emerged. In 
addition, these teachers helped to solidify the treatment fidelity because they 
were key researchers in the development and delivery of the unit of instruction. 

The unit of instruction included four lessons, with specific content and 
activities. Although the units of instruction include activities that may seem 
similar to previously published curricula, the teachers developed the units of 
instruction independent of established materials because the focus of the units of 
instruction was on constraints, optimization, and predictive analysis. Other than 
the first lesson and activity (see below), teachers were permitted to teach the 
remaining lessons in any order. The unit of instruction was scheduled to be 
completed in 20 class sessions. Some of the teachers completed the unit prior to 
the 20 days, while other teachers went beyond the targeted number of days. 

Each lesson had a student version and teacher version. The teacher version of 
each lesson included supplemental materials, including presentation materials 
specifically designed to address the key concepts being delivered in the unit. 
Grading rubrics were provided to the teachers and students for each lesson. 
Below is a description of each lesson and activity. 

The first lesson and activity (treatment) that the students (research 
participants) completed during this study to introduce COPA was called 
“Volume Barge.” In this lesson and activity, students were challenged to design 
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and create a barge-type artifact made from one piece of 8.5” x 1 1” laminated 
card stock capable of holding the most weight before sinking; the barge had to 
be a rectangular shape. Students used volume calculations to optimize the best 
design based on the constraints. In a competitive style format, students graphed, 
using Microsoft Excel, the entire class performance to determine the winner. 
Using calculus based concepts, an optimum volume value was established, 
which set the standard for optimizing the design. This lesson/activity was 
deemed the favorite by most of the students from the eight schools, largely due 
to its competitive nature. 

A second lesson and activity the students completed was related to energy 
efficiency. During this lesson, students used mathematical formulas and existing 
data to determine R and U values for insulated wall cavities. Each group of 
students was provided with four completed wall sections, each having a 12” x 
12” opening constructed with 2” x 4” and 2” x 6” framing materials. Three of 
the wall cavities were filled with different insulating materials, while the fourth 
section was left empty. The students were challenged to calculate the efficiency 
versus cost in a life-cycle approach, to determine the most optimal choice for 
insulation based on an average daily temperature and cost per thermal unit. Each 
completed wall section (four in all to create a square) was covered with a sheet 
of plywood, and a 100 watt incandescent lamp was placed in the center. Using 
an infrared heat-sensing device, the students were able to determine insulation 
efficiency. From the experiences learned from this activity, the students had to 
use predictive analysis to optimize the life cycle costs of construction and 
building ownership over 5, 10, 15, and 25 year periods for a 2000 square foot 
structure. Students used Microsoft Excel to graph their results and present them 
to the class. This lesson and activity was classified as the second favorite of 
most of the students from the eight schools, largely because it involved a hands- 
on experience. 

The third lesson and activity the students completed utilized a pre- 
fabricated golf ball launching device made from PVC and wood framing 
materials. Students used predictive analysis techniques to accurately launch a 
golf ball from a specific height and angle to a specific end distance. Students 
had to understand vectors, laws of motion, and energy to succeed. Students first 
predicted (non-analytic) how far the golf ball would travel and then used 
mathematical formulas to analyze how and where the ball would travel. In 
addition, students quickly learned that there exists an efficiency factor and that 
no machine or mechanism is 100% efficient. Students graphed their results and 
presented their findings to the class. This lesson and activity was rated as the 
third favorite by most of the students from the eight schools involved in this 
study, largely because the students could see the mathematical calculations in 
action in observing where the ball should and would land. 

The fourth lesson and activity the students completed dealt with identifying 
where and how mechanical energy is used and lost in their school. As an 
introductory activity to this lesson, students were provided with four different 
types of light (incandescent, fluorescent, halogen, and LED ) to examine their 
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efficiency. After classroom discussion regarding the cost, life-span, and energy 
used from these different sources of light, students became engaged in an 
activity in which they had to locate four different sources of mechanical energy 
in their school to determine their efficiency. Students created a proposal that 
outlined where mechanical energy is being lost throughout their school and how 
they would use the wasted energy to complete productive work in other 
applications. Students used Newton scales, stop watches, and tape measures to 
determine mechanical energy (i.e., force, distance, and time). Each potential 
solution that students determined also included a wattage factor. During the 
presentation, students discussed their data collection methods, design solutions, 
the constraints associated with each design solution, and how each design would 
be optimized. Overall, students enjoyed this lesson and activity, but felt because 
it lacked a hands-on (building) approach, their engagement was not as high. 

Instrumentation 

The research team and a technology education teacher, who has a 
mechanical engineering degree, developed the test instrument used in this study. 
The development of the test instrument was guided by a review of the literature 
related to the engineering concepts: constraints, optimization, and predictive 
analysis. The thirty-item test instrument was developed to target the three 
concepts across three levels of Bloom’s taxonomy. For the purpose of this 
study. Bloom’s framework was reduced to three levels: comprehension, 
application, and analysis/synthesis. Comprehension included Bloom’s 
Knowledge and Comprehension categories, application included Bloom’s 
Application category, and analysis/synthesis combined Bloom’s Analysis, 
Synthesis, and Evaluation categories (Dalton, & Smith, 1986). For example, 
questions written at the comprehension level used verbs offered by Dalton and 
Smith, including explain, predict, or discuss. 

Ten items were developed to target each of the three engineering concepts 
and were spread relatively evenly across the three levels of Bloom’s taxonomy. 
At the comprehension level there were four constraint, four optimization, and 
four predictive analysis questions. At the application level there were three 
constraint, three optimization, and three predictive analysis questions. At the 
analysis/synthesis level there were three constraint, three optimization, and three 
predictive analysis questions. 

The instrument was subjected to three revision cycles before a final version 
was established. The first cycle consisted of an internal review by the 
researchers. The second cycle consisted of a pilot test that was administered to a 
cohort of practicing technology education teachers during the summer 
professional development experience. This was done to determine whether the 
instrument was at the appropriate level of difficulty for secondary students and 
to identify any problematic questions. Their estimates were based on many years 
of experience of working with secondary level students. The research team and 
the technology education teacher then later refined potentially problematic 
questions. The third cycle consisted of an expert panel review where content 
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validity was verified by sending the instrument to a panel of engineering and 
technology education professors and practitioners. These individuals completed 
a review of the instrument and a survey asking whether the questions measured 
an understanding of the three concepts at the different levels. Based on the 
feedback from the expert review panel, the test was further refined by the 
research team. This process resulted in an instrument containing thirty items that 
were believed to be at the appropriate difficulty level that measured an 
understanding of COPA at the three different levels. The same instrument was 
used for both the pretest and posttest. The reliability of the test instrument using 
Cronbach's coefficient alpha was r = .782. 

Sampling Procedures 

A purposive sampling frame was utilized for this study, so the same 
teachers who developed the instructional materials were able to deliver the 
actual instruction in their classes to the study participants. Eight of the 
technology education teachers who participated in the NCETE professional 
development sessions at ISU recruited the students from their schools. 
Recruitment was conducted in nine intact technology education classes, since 
one teacher was able to recruit participants from two separate classes. Initially, 
124 high school level technology education students agreed to participate in the 
study. However, as a result of attrition, only 1 14 (n = 1 14) students remained in 
the study at the time of the posttest. Within the final population there were 102 
male and 12 female students. 

In order to assess if there were significant differences between the subjects 
who remained in the study and those who did not, a one-way ANOVA at the a 
=.05 level was conducted using the pre-test scores as the dependent variable. 
The results suggested that there were not significant differences between the 
two groups in terms of test scores F( 1,123) = .04, p > .05. Table 1 illustrates the 
grade level and number of research subjects (students) per grade level involved 
in this study. It should be noted that all high school teachers and students were 
from Illinois schools. 

Table 1 


Grade level of participants 



/ 

Percent 

9 th Grade 

14 

12.4 

10 th Grade 

17 

15.0 

11 th Grade 

48 

42.5 

12 th Grade 

34 

30.1 


n= 114 


Current or previous mathematics course involvement of students in the 
sample included 72% in Algebra I, 63% in Algebra II, 71% in Geometry, 46% 
in Trigonometry, 34% in Pre-Calculus, and 6% in Calculus. The breakdown of 
science classes completed or that were presently being taken by the high school 
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students was 47% in General Science, 76% in Biology, 57% in Chemistry, and 
36% in Physics. 

Focus Groups 

A formal, directive, structured focus group was deemed most appropriate 
for this study (Fontana & Frey, 2005). As Morgan (2002) argued, by 
systematically approaching the focus group interviews a methodological 
continuity is created so as to better assess the outcomes. Within twenty-four 
hours after completing the posttest, a focus group of systematic selection 
procedure was used to identify students from the eight high schools who 
participated in the study. Using the alphabetized course roster for each of the 
nine classes, every third student was selected to potentially participate in focus 
group. However, if that student did not wish to participate or had not submitted 
a consent form, that student was not selected for the sample. In a few cases, 
every third student resulted in a sample size of less than six, so two cycles of 
every third student, beginning with the last student selected, was conducted. For 
one school, the class size was small enough to conduct a focus group with all of 
the students. For two other schools, only a few students submitted consent forms 
or agreed to participate in the focus group, so all of those students participated 
in the focus group. A total of nine students from each school were selected to 
participate in the focus groups. The first six students selected were targeted as 
the primary participants, and three students were selected as alternates in case of 
absences or withdrawals. A total of eight schools and nine separate classes were 
involved with the focus group (n = 54). Of the 54 students selected for the focus 
groups, 47 were males, 7 were females. 

Each focus group, lasting no longer than forty-five minutes, was guided by 
a script of fifteen questions that were divided into three categories: appeal 
questions, probing questions, and suggestions for improving the unit of 
instruction. Each focus group was conducted by two members of the research 
team. One of the researchers asked the questions and the other researcher acted 
as the scribe. In some cases, both the interviewer and scribe asked the students 
questions. Each focus group session was digitally recorded and saved as an MP3 
file. Each of the researchers independently listened to each focus group session 
and developed synthesis paragraphs. 

Procedures 

This study was conducted in two phases. Phase One was the development 
of the 20-class session unit of instruction to be delivered to high school level 
technology education students. Phase Two consisted of delivering and assessing 
the unit of instruction with the participants of this study. Phase One began 
during the Summer 2006 technology education session at ISU. Twelve high 
school level technology education teachers attended a five-day professional 
development session to develop the unit of instruction to integrate COPA in 
their technology education courses. Some of the teachers in this experience had 
attended ISU’s professional development sessions the previous year. Those 
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sessions were also focused on integrating engineering concepts into high school 
level technology education curricula. 

During the first two days of the summer 2006 session, the teachers 
participated in presentations by technology education faculty members from 
ISU, an engineering professor from the University of Illinois at Urbana- 
Champaign (UIUC), NCETE doctoral fellows from UIUC; and science, 
mathematics, and technology education teachers from the public high school 
system. These presentations included a review of COPA, Wiggins and 
McTighe’s backward design process (Wiggins, & McTighe, 2005), and 
activities that emphasized the COPA concepts. 

During the third and fourth day of the workshop, the teachers working in 
groups of four developed the unit of instruction using the backward design 
process (Wiggins & McTighe, 2005). As discussed in the instruction 
development section, each group of teachers developed artifacts for each of the 
different activities. Concurrently, the NCETE doctoral fellows, in conjunction 
with the technology education teacher from the high school system, began 
developing the test instrument to assess students understanding of COPA. 

On day five, the initial instrument was pilot tested with the high school 
teachers and scored. The results were shared with the teachers in order to obtain 
feedback regarding content and construct validity, and appropriate floor and 
ceiling height for high school level students. Additionally, the groups of 
teachers presented their activities to the session participants in order to obtain 
feedback that could be used to later enhance the activities. After the conclusion 
of the experience, the eight teachers who would actually be able to deliver the 
unit of instruction were asked to continue with the refinement and final 
development of the activities. 

Between August and October, final lessons and activities related to Phase 
One were completed: (a) Institutional Review Board protocol approval was 
obtained from ISU and the UIUC, as well as from the individual high schools, 
(b) the test instrument was further refined after obtaining feedback from 
engineers, technology education professors, and practicing teachers, and (c) the 
eight teachers returned to ISU on October 5, 2006 for a one-day session to 
deliver the finalized units of instruction to the participating teachers involved in 
the study in order to receive formative feedback. Additionally during this time, 
technology education pre-service teachers enrolled at ISU began constructing 
the artifacts that were needed for the activities and pre-assembled these into kits 
that were sent to the participating high schools. The use of pre-assembled kits 
was deemed necessary in order to maintain treatment fidelity by making certain 
that all sites were using identical materials. In addition to uniformity provided 
by the pre-assembled kits, the process reduced the total time required to 
implement the study. This time element was an important factor, since teachers 
were injecting the research unit into their regular semester’s curriculum. 

Phase Two began during the first week of October 2006. Since the high 
school students were under 18, both parental consent and student assent were 
required. The technology education teachers distributed the appropriate forms to 
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the participants, signatures were obtained, and the forms were returned to their 
teachers who then forwarded the forms to the research team. The participants 
were then administered the pre-test that consisted of the 30 items related to 
measuring COPA at the three. Bloom-based levels of understanding. 

The delivery of the unit instruction by the teachers began between the 
second and third week of October 2006. Because the teachers were working 
within the limitations of their existing curriculum, it was not feasible for all sites 
to begin delivery of the unit of instruction on the same day. The unit of 
instruction was delivered over the span of 20 class sessions. Immediately 
following the conclusion of the last class session, the posttest was administered 
to the participants and returned to the research team. As previously noted, the 
pretest and post-test were identical. 

Findings and Discussion 

Consistent with the mixed model research design, data analysis consisted of 
quantitative and qualitative components. With the quantitative component, a 
series of dependent groups /-tests were conducted to explore differences 
between the pretest and post-test. Student performance related to selected 
demographic variables was also examined. Qualitative data analysis consisted of 
examining the transcripts of post-instruction focus groups, which were 
conducted by the research team to explore students' understanding of core 
concepts and the efficacy of the study’s unit of instruction. 

Quantitative Data Analysis 

A series of dependent groups /-tests were conducted to compare pretest and 
post-test scores. The initial analysis, which was conducted on the composite test 
scores, was followed by separate analyses of the instrument’s three dimensions 
(i.e., constraints, optimization, and predictive analysis). Student scores consisted 
of the number of items answered correctly of the instrument's 30 questions. A 
significant composite score gain of 3.22 was obtained between the pretest and 
post-test (see Table 2). 

Table 2 


Dependent groups t-test for composite test score (n = 114) 



Mean 

Std. Dev. 

t 

P 

Pre-Test Score 

m 


8.604 

.000 

Post-Test Score 

■H 

■EM 




Total items on the test = 30 


While the gain scores were statistically significant, the overall percentage of 
items answered correctly was somewhat disappointing. Based on focus group 
discussion and interaction with participating teachers, low composite test scores 
were attributable to several factors. Among these factors were perceived test 
difficulty and voluntary participation in the study, where students were informed 
that test results would not be counted in their semester grades. In spite of 
relatively low test scores, focus group discussion, however, indicated that 
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students clearly were able to identify the core engineering concepts selected for 
the study. In the aggregate, they also possessed a relatively sophisticated 
understanding of the interrelationship among the concepts. Focus group results 
suggest that the instruction may have been more effective than what the test 
scores indicate. 

In a focus group conducted with participating teachers, there was broad 
consensus that the test results were negatively impacted by the structure of the 
test. This included its high readability and conceptual levels as well as 
difficulties with knowledge transfer between examples used in the instrument 
and activities used during instruction. In spite of efforts made to validate the 
instrument during the planning stages of the study, it was clear that the study’s 
outcomes were influenced by these assessment issues. In spite of these issues, 
the teachers indicated that their students’ understanding of COPA concepts was 
clearly achieved as a result of the study. 

Separate dependent groups /-tests were also performed on the instrument’s 
sub-scores (i.e., items assessing the three core engineering concepts). Mean 
score gains ranging from approximately 1 - 1 Vi items were obtained, with the 
highest gain score on the predictive analysis dimension. All three gain scores 
were statistically significant (see Table 3). 

Table 3 


Dependent groups t-test for core concept dimensions (n - 114) 



Mean 

Std. Dev. 

t 

P 

Constraints (10 items) 





Pre-Test 

6.13 

2.106 

4.687 

.000 

Post-Test 

6.98 

2.018 



Optimization (10 items) 





Pre-Test Score 

4.22 

1.718 

5.513 

.000 

Post-Test Score 

5.12 

1.942 



Predictive Analysis (10 items) 





Pre-Test Score 

4.39 

2.151 

7.478 

.000 

Post-Test Score 

5.85 

2.019 




n= 114 


The instrument’s design also included the development of items at three 
levels of complexity along Bloom’s taxonomy of cognitive skills 
(comprehension, application, and analysis/synthesis). As could be anticipated, 
the highest net score gain occurred at the comprehension level (2.07) and the 
lowest score gain was obtained at the analysis/synthesis level (.53) (see Table 
4). 

Table 4 


Dependent groups t-test for conceptual difficulty levels (n - 114) 



Mean 

Std. Dev. t 

P 

Comprehension (12 items) 
Pre-Test 

6.06 

2.557 9.277 

.000 

Post-Test 

8.13 

2.533 
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Application (9 items) 


Pre-Test Score 

4.62 

1.962 

3.613 

.000 

Post-Test Score 

5.25 

2.072 



Synthesis (9 items) 





Pre-Test Score 

4.05 

1.645 

2.893 

.005 

Post-Test Score 

4.58 

1.499 




As noted above, focus group discussion indicated that the level of students’ 
understanding may exceed that suggested by the test score data. As stated 
earlier, student scores were relatively low, but subjects' understanding of the 
three core concepts was evident in the focus groups. For example, a common 
observation of students was how the three engineering concept dimensions 
interact with one another in real world engineering design situations. 
Specifically, students commented on how optimized engineering designs 
routinely require tradeoffs among constraints (e.g., it is not possible to optimize 
all constraints; constraints tend to compete with one another). 

The data were also analyzed to examine the possible effects of selected 
demographic factors on student learning. These factors included gender, 
ethnicity, and level of mathematics and science courses. Analysis of variance 
procedures, which were conducted on each of the variables, detected no 
statistically significant differences. Non-significant differences on the selected 
demographic variables are encouraging. Given the well-documented concern 
about female and minority involvement in scientific and engineering careers, 
this study’s results indicate that gender and ethnic differences may be 
minimized or reduced in controlled instructional situations. In other words, 
gender and ethnically-sensitive instructional design may facilitate learning of 
engineering concepts in ways that minimize demographic differences. The 
results of this study are encouraging, given the concern of many technology 
educators that the growing emphasis on engineering could reduce participation 
of a broad spectrum of students. 

Although not a research hypothesis of this study, the gender composition of 
the sample is presented in Table 5. The analysis indicated that there were no 
significant gender differences. 

Table 5 


Independent groups t-test for gender on post-test concepts 
(males = 101, females = 12, = 114 total) 



Mean 

Std. Dev. 

t 

P 

Constraints (10 items)) 





Pre-Test 

6.98 

2.044 

.032 

.974 

Post-Test 

7.00 

1.859 



Optimization (10 items) 





Pre-Test Score 

5.15 

1.987 

.387 

.699 

Post-Test Score 

4.92 

1.564 



Predictive Analysis (10 items) 





Pre-Test Score 

5.82 

2.036 

.420 

.675 

Post-Test Score 

6.08 

1.929 
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Qualitative Data Analysis 

All of the focus group students commented on the amount of mathematics 
and science that was included within the unit of instruction and activities; 
predictive analysis throughout all of the activities was the least favorite aspect 
of the twenty-day unit of instruction. With the exception of one focus group, all 
of the students knew that constraints, optimization, and predictive analysis 
(COPA) were the key concepts being taught throughout the unit of instruction 
and provided examples of how and when they learned these concepts. As 
discussed in the treatment section, each lesson and activity targeted one or more 
of the key concepts. A majority of the students defined optimization as “the best 
solution to a problem, balancing trade-offs between competing factors.” 

Students defined predictive analysis as the “mathematical or scientific equations 
that are used before the artifact or problem is completed.” Students defined 
constraints as “the boundaries for what you can do and the parameters you have 
to stick to.” 

Of the three COPA concepts, predictive analysis was the most difficult to 
understand for the majority of the high school students and constraints was the 
easiest. A small minority of the students who completed the focus groups 
identified the COPA concepts as interconnected. Most students used an analogy 
of the scientific method to COPA. A majority of students were not familiar with 
optimization and predictive analysis before the treatment, but through their prior 
or existing technology education courses were familiar with constraints. Almost 
all students commented that they take technology education courses because 
they are fun and activity-based, not mathematics or science -based. 

All of the focus group students rated the “Volume Barge” activity as their 
favorite. This activity focused on the concept of optimization. Furthermore, the 
activity challenged students to compete against their classmates for the best 
barge. A majority of the focus group students wanted more open design 
activities that were similar to the barge activity. 

Almost all of the focus group sessions revealed that students wanted a 
launching device, similar to a pneumatic powered device instead of the golf ball 
launching device that was used in the unit of instruction; students wanted a 
“boom” effect rather than the gravity fed device provided. Students seemed to 
like the wall insulation activity because it was more hands-on than the other 
activities within the unit. 

Overwhelmingly, the students in the focus groups commented on how 
mathematics and science concepts taught throughout the unit were better 
understood when they were connected to solving a problem or building an 
artifact. Students commented that they did not understand mathematics and 
science in their stand alone courses. They also commented on the positive nature 
of including most of the formulas they would need to solve for problems within 
student handouts or embedded within the activities. However, students 
commented that the theory of mathematics and science does not always translate 
into a properly working artifact. 
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The last question to each focus group was whether or not this unit of 
instruction had influenced them to pursue post-secondary studies and a career in 
engineering or a related field. There was no indication that after completing the 
twenty-day unit of instruction that a positive or negative influence existed. 

Conclusions and Recommendations 

Based on the results of this study, some conclusions become apparent. It is 
clear that student learning was achieved as a result of students’ participation in 
the engineering design unit of instruction. While mean score gains from pre-test 
to post test were modest, they did indicate significant improvement in 
understanding of COPA concepts. Given the lack of significant gender, ethnic 
group, and mathematics/science background performance differences, the study 
also indicates that engineering concepts can be successfully delivered to a broad 
spectrum of students. These preliminary results are important since many 
technology educators are concerned that an engineering curricular focus might 
appeal only to a more academically capable subset of the technology education 
student body. 

Based on focus group discussions with students and teachers, some 
important factors emerged related to how engineering concepts were delivered 
in this study. These factors have important implications for future research, 
curriculum development, and professional development. One key factor has to 
do with an overt shift from procedural/activity-based curriculum and instruction, 
which as been typical for technology education, to an overt concept-based 
focus. The importance of this shift certainly extends beyond this research study 
or engineering curriculum. In this era of standards-based instruction, the 
technology education field must learn how to balance the historical appeal of 
engaging activities with curriculum development that is specifically designed to 
teach concepts (standards). While students generally indicated that they enjoyed 
the study’s activities, they also reported that they would have preferred to have 
actually constructed more of the devices used in the study rather than having 
them pre-constructed in order to meet the time and treatment fidelity constraints 
of the study. 

Another significant challenge of research of this type has to do with the 
constraints involved with informed consent research. Focus group results 
indicate that student motivation to perform well in the study was eroded by their 
awareness that the test outcomes would not be included in their semester grades. 

Another important factor that emerged from the study had to do with the 
challenges associated with developing high quality, authentic assessments of 
COPA concepts. The outcomes of the study indicate that the test instrument was 
capable of detecting student learning at the various levels of conceptual 
difficulty. The psychometric properties were also sound, with acceptable levels 
of reliability and validity. Teacher involvement in the development and 
validation of the instrument used in this study was designed to ensure its 
appropriateness, including appropriate level of difficulty. However, focus group 
feedback indicated that students found the items to be demanding both in terms 
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of reading level and conceptual load. While the multiple choice format provided 
objective data, future research should include more diverse and authentic 
formats. 

Nonetheless, the findings of this study also indicated that there are specific 
areas of need in order to better develop these engineering concepts. For 
example, existing and pre-service technology education teachers need to be 
better equipped to develop and teach instruction focused on engineering design 
concepts. In particular, professional development focused on preparing 
technology education teachers to develop and teach instruction that is both 
concept-driven and activity-oriented is an area of need. Historically, technology 
education has focused primarily on procedural knowledge through hands-on 
activities that focus primarily on artifacts. In order to integrate engineering 
concepts within technology education, technology education teachers need to 
develop pedagogical skills that include more focus on conceptual knowledge 
and the processes involved in engineering design, which includes the ability to 
apply mathematical and scientific knowledge. 

Another area of need is the development of sound curriculum, activities, 
and assessments that target engineering design concepts. The instruction and 
activities developed for this study appear to have done an adequate job relaying 
the concepts to the students. However, with more refinement and focus these 
and similar activities could be used to teach engineering concepts even more 
effectively beyond the twenty-session research treatment. For example, separate 
units on constraints, optimization, and predictive analysis may help students 
better understand these engineering concepts. These activities need to maintain a 
hands-on component, which is an area of strength for technology education, 
because it appears to be a key to student motivation. In addition, authentic 
assessments need to be developed to assess student understanding. As revealed 
in this study, there are limitations to using tests to assess student learning of 
engineering concepts, especially at the analysis/synthesis level. Authentic 
assessments targeted at assessing student’s understanding need to be developed 
to gauge student learning. 
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